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_3The  Navy  plans  to  buy  complex  avionics  computer  systems,  and  related 
communications  equipment  and  sensors,  for  submarine  patrol  aircraft. 
This  acquisition,  designated  the  Update  IV  Program,  is  intended  to  pro¬ 
vide  the  Navy  with  the  capability  to  locate,  identify,  and  attack  the 
expected  threat  of  more  quiet  submarines/  Between  September  1990  and 
May  1993,  the  Navy  plans  to  buy_28-o£tfie  avionics  systems  at  a  cost  of 
$49§jniHionrAltK5nth  its  plans  are  uncertain,  the  Navy’s  total  purchase 
/''could  be  for  up  to  240  systems  at  a_cost.oLabout-$2,.LblIlion. 

This  report  responds  to  your  offices’  October  1989  request  to  review  the 
Update  IV  Program,  and  is  part  of  your  overall  request  to  reviewihe 
Department  of  Defense’s  acquisition  of  computer  systems  embedded  in 
weapon  systems.  Ottifybjectives  were  to  determine  whether  (1)  the  Navy 
plans  to  adequately  test  the  avionics  computer  systems  before  buying 
them,  and  (2)  Navy  management  oversight  of  these  computer  systems 
has  occurred!  A  detailed  discussion  of  our  objectives,  scope,  and  method¬ 
ology  is  contained  in  appendix  I. 


■The  Navy  is  taking  a  high  risk  approach  in  acquiring  a  new  and  complex 
computer-based  avionics  system  for  its  patrol  aircraft.  Although  the 
Navy  originally  planned  to  thoroughly  test  this  system  before  buying 
more  than  four,  program  delays  led  the  Navy  to  postpone  complete 
testing.  This  is  clearly  contrary  to  (1)  Defense  policies  which,  when  fol¬ 
lowed,  should  be  effective  in  mitigating  computer  system  development 
risks  and  (2)  the  principle  of  “fly  before  you  buy.” 


^DISTRIBUTION  STATEMENT^ 

Approved  for  public  release; 

' _ Distribution  Unlimited 
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Specifically,  the  Navy  plans  to: 

•  Continue  developing  software  (i.e.,  coding)  before  it  has  approved  the 
design  specifying  what  the  software  should  do  and  how  well  it  should  do 
it; 

•  Buy  28  of  the  avionics  systems  before  all  testing  is  successfully  com¬ 
pleted;  and 

•  Use  a  model  of  one  of  the  system’s  processors  during  testing  that  is  not 
an  accurate  representation  of  the  final  version. 

The  Navy  is  planning  to  follow  this  high  risk  approach  because  it 
believes  that  any  further  delays  will  cause  it  to  miss  fixed-price  contract 
option  deadlines  and  increase  contract  costs.  The  Navy,  however,  has 
not  prepared  any  detailed  analysis  to  support  its  contention  that  con¬ 
tract  costs  will  increase.  In  addition,  the  Navy’s  position  fails  to  consider 
the  costs  of  buying  28  systems  that  may  not  work  as  intended  and  may 
require  expensive  fixes,  assuming  they  can  be  fixed.  As  we  previously 
reported,  it  can  be  six  to  ten  times  more  costly  to  correct  a  software 
problem  after  a  system  in  placed  in  operation  than  it  is  during  early 
system  development. 

We  recognize  that  adhering  to  Defense  policies  might  increase  acquisi¬ 
tion  costs.  But  possible  cost  increases  do  not  justify  spending  almost 
$500  million  on  a  system  that  has  not  been  thoroughly  tested.  If  the 
Navy  finds  that  missing  contract  option  deadlines  will  be  prohibitively 
expensive,  it  must  decide  whether  this  avionics  system  is  affordable. 


Background 


In  February  1985,  the  Navy  began  the  $2.1  billion  Update  IV  Program  to 
provide  submarine  patrol  aircraft  with  modem  avionics  computer  tech¬ 
nology  and  sensors.  These  long  range,  land-based  patrol  aircraft  are 
deployed  globally  to  find,  identify,  and  attack  new  classes  of  very  quiet 
enemy  submarines.  The  Navy  planned  to  install  the  new  avionics  com¬ 
puter  system  on  108  existing  P-3  aircraft  and  an  estimated  125  new  P-7 
aircraft.1  However,  the  Navy  has  since  terminated  the  P-7  aircraft  pro¬ 
gram,  and  is  now  considering  other  alternatives  such  as  buying  more  P-3 
aircraft  or  reducing  the  number  of  avionics  systems  to  be  bought. 

The  Update  IV  Program  includes  computer  systems  that  process  and  dis¬ 
play  sensor  data,  and  control  aircraft  sensor,  communication,  naviga¬ 
tion,  and  armament  subsystems.  In  1987,  the  Navy  awarded  an  Update 


'The  remaining  7  of  the  240  total  avionics  systems  are  for  engineering  development  modeling. 
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IV  system  integration  contract  having  multiple  contract  production 
options,  including  an  initial  four  that  are  fixed-price.  The  system’s 
software  is  currently  being  developed  (i.e.,  coded)  and  undergoing  early 
laboratory  testing.  The  first  contract  option  deadline  is  September  1990. 
Table  1  shows  development  milestones,  the  first  three  contract  option 
quantit  ies,  and  funding  requirements. 


Table  1:  Development  Milestones,  First 
Three  Contract  Options,  and  Funding 
Requirements 


Dollars  in  millions 

Funding  Requirements* 

Milestones 

Quantity 

Production 

Support 

Total 

Option  1 

September 

1990 

4 

$68.3 

$21.2 

$89.5 

Option  2 

May  1992 

12 

$153.9 

$41.1 

$195.0 

Option  3 

April  1993 

12 

$174.3 

$37.6 

$211.9 

Totals 

28 

$396.5 

$99.9 

$496.4 

“Funding  requirements  have  not  been  revised  to  reflect  any  near-term  impact  of  terminating  the  P-7 
program. 


Navy  Did  Not  Specify 
Software 

Requirements  Before 
Development 


The  Update  IV  system  integration  contract  requires  the  contractor  to 
prepare  detailed  functional  and  performance  specifications  for  each 
software  subsystem.  According  to  Defense  software  development 
policy,2  such  specifications  are  necessary  to  establish  a  requirements 
baseline  for  detailed  software  design  and  development.  Contrary  to  this 
policy,  however,  the  Navy  is  allowing  the  Update  IV  contractor  to 
develop  software  before  subsystem  specifications  are  completed  and 
approved.  The  Defense  Department  has  reported3  and  we  agree  that 
failure  to  define  complete  specifications  before  developing  software  may 
not  only  jeopardize  software  quality,  but  can  also  increase  development 
costs  and  delay  project  completion.  This  concern  was  raised  in  a  risk 
analysis  prepared  by  the  Navy  laboratory  monitoring  the  contractor’s 
performance.  As  stated  in  the  analysis,  developing  the  software  that 
will  perform  some  of  the  system’s  mission  functions  (e.g.,  communica¬ 
tion  and  navigation  processing,  nonacoustic  sensor  management,  search 
stores  and  weapons  management,  and  inflight  performance  monitoring) 
at  the  same  time  that  software  requirements  are  being  specified  affords 


2Military  Standard  2167A,  Defense  System  Software  Development,  February  1988. 

3 Proceedings  of  the  Joint  Logistics  Commanders  Joint  Policy  Coordinating  Group  on  Computer 
Resource  Management  (Aug.  1979). 
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minimal  time  for  much-needed  design  validation  and  greatly  increases 
the  risk  that  the  software  will  be  unacceptable. 

As  of  June  1990,  the  Navy  had  not  approved  most  subsystem  specifica¬ 
tions  for  the  Update  IV  avionics  system,  yet  the  contractor  had  written 
about  80  percent  of  the  software  and  had  started  testing  to  determine 
whether  individual  units  of  code  performed  their  functions.  Although 
draft  specifications  exist,  the  Navy  has  approved  or  conditionally 
accepted  only  a  third  of  them  because  the  remaining  two  thirds  either  do 
not  meet  the  requirements  of  the  system  level  specifications  or  have  yet 
to  be  evaluated.  Consequently,  the  avionics  software  is  being  coded 
against  a  moving  baseline,  and  the  Navy  currently  lacks  firm  criteria  for 
testing  and  contract  acceptance.  This  has  not  only  resulted  in  an  esti¬ 
mated  3-month  delay  in  software  testing,  but  more  importantly  could 
result  in  (1)  additional  coding  costs  once  the  software  design  is  finalized 
or  (2)  a  system  which  does  not  meet  all  requirements  set  forth  in  the 
system  specification. 

We  reviewed  the  status  of  the  Update  IV  subsystem  specifications  as  of 
June  1990,  and  found  that  the  Navy  had  approved  only  3  of  93  specifi¬ 
cations.  Of  the  remaining  90  specifications,  30  had  received  conditional 
approval  pending  incorporation  of  minor  comments,  17  were  unaccept¬ 
able  and  rejected  because  they  did  not  meet  requirements  in  the  overall 
system  specification,  and  43  were  still  under  review.  Examples  of 
rejected  specifications  include  those  for  the  tactical  mission  subsystem4 
and  the  systems  management  software,6  without  either  of  which  the  avi¬ 
onics  system  could  not  accomplish  the  antisubmarine  warfare  mission 
objective.  Although  approval  of  all  subsystem  specifications  was 
targeted  for  August  1990,  program  officials  stated  that  approval  will 
not  occur  before  September  1990. 


Adequate  Testing  Will 
Not  Occur  Prior  to 
Production  Decisions 


The  Navy’s  planned  testing  of  the  Update  IV  Program  will  not  provide 
reasonable  assurance  that  this  computer-based  avionics  system  is  ready 
for  full  rate  production.  Testing  validates  that  a  system  meets  its  func¬ 
tional  and  performance  requirements  and  can  effectively  perform  the 
intended  mission.  For  the  Update  IV  Program,  testing  is  particularly 


4This  subsystem  provides  environmental  analysis  aids,  correlation,  tactical  execution,  tactical  and 
nonacoustic  situation  management,  classification,  and  tactical  planning  aids. 

6This  software  is  responsible  for  recording,  data  base  management,  displays  and  controls,  and  real¬ 
time  executive  management  functions. 
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important  because  the  system’s  software  is  estimated  at  about  one  mil¬ 
lion  lines  of  Ada  code,6  and  the  Navy  laboratory  monitoring  the  Update 
IV  Program  considers  Ada’s  use  to  be  a  high  risk  because  relatively  little 
experience  has  been  gained  with  Ada  in  stringent,  real-time  environ¬ 
ments.7  Additionally,  this  software  performs  critical  mission  functions. 
For  example,  the  avionics  system’s  software  includes  sophisticated  algo¬ 
rithms,8  such  as  multisensor  target  tracking  and  decision  aids,  to  help 
crew  operators  work  with  the  massive  number  of  sensor  data  inputs 
associated  with  locating,  tracking,  and  targeting  submarines.  The  cor¬ 
rectness  and  effectiveness  of  such  complex  software  systems  is  ascer¬ 
tained  by  thorough  testing.  Failure  to  conduct  rigorous  testing  greatly 
increases  the  possibility  of  producing  and  deploying  a  system  that  fails 
to  meet  its  mission  requirements. 

Computer  system  testing  is  incremental  and  can  be  viewed  as  having 
two  major  components — developmental  testing  and  operational  testing. 
Generally,  the  purpose  of  developmental  testing  is  to  determine  whether 
a  system  meets  the  requirements  in  its  specifications,  while  operational 
testing  determines  whether  a  system  that  has  been  designed  according 
to  its  specifications  meets  mission  requirements.  More  specifically,  early 
developmental  tests  focus  on  whether  individual  software  modules  per¬ 
form  as  required  in  the  specification.  Later  developmental  testing 
addresses  whether  and  how  well  the  integrated  modules  perform 
required  functions  (i.e.,  how  fast,  how  reliable,  how  accurate,  how 
often)  in  a  laboratory  that  realistically  simulates  an  operational  setting. 
Following  this  laboratory  integration  testing,  the  complete  system  is 
tested  with  actual  users  in  a  true  operational  setting.  This  operational 
testing  is  sometimes  conducted  in  two  phases,  with  the  first  phase 
showing  the  system’s  “potential”  mission  effectiveness  and  justifying 
initial  rate  production  quantities,  and  the  second  phase  demonstrating 
the  system’s  ability  to  meet  the  mission  requirement  and  justifying  full 
rate  production.  The  above  described  testing  progression  emphasizes  the 
benefits  of  finding  problems  early  in  the  development  process,  when 
they  are  cheaper  to  correct. 


“Ada  is  a  relatively  new  high-order  language  designed  for  use  in  real-time  computer  systems. 

7Such  systems  must  be  able  to  obtain  data  from  an  activity  or  process,  perform  computations,  and 
respond  quickly  enough  to  affect  the  outcome  of  that  activity  or  process.  Depending  on  the  applica¬ 
tion,  a  response  may  be  required  in  seconds  or  in  milliseconds.  Aircraft  avionics  use  real-time  com¬ 
puter  systems. 

8Well-defined  steps  for  solving  a  problem. 
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Developmental  Testing 
Hampered  by  Contractor 
Laboratory  Test  Facilities 


The  contractor  is  responsible  for  developing  two  software  integration 
laboratories,  one  for  testing  the  avionics  system’s  acoustic  capabilities 
and  one  for  testing  its  nonacoustic  capabilities.  These  laboratories 
include  computer  hardware  and  software  that  simulate  the  avionics 
system’s  operational  conditions  and  provide  an  environment  for  testing 
the  system. 


The  contractor  is  currently  more  than  1  year  late  in  developing  the  two 
laboratories.  As  a  result,  the  extent  of  laboratory  testing  that  can  be 
performed  before  the  first  production  options  expire  has  been  reduced 
considerably.  Moreover,  the  Navy’s  technical  revv-w  of  the  contractor’s 
laboratory  facilities  has  raised  some  doubt  as  to  wh<.  her  the  laborato¬ 
ries’  simulation  programs  accurately  simulate  the  mission  environment. 
For  example,  the  review  questions  whether  the  laboratory  simulation 
includes  a  realistic  number  of  targets. 


Operational  Testing  Will 
Not  Be  Completed  Before 
Systems  Are  Bought 


Navy  policy9  requires  full  operational  testing  before  a  system  enters  full 
rate  production.  Additionally,  the  National  Defense  Authorization  Act 
for  Fiscal  Years  1990  and  1991,  dated  November  29, 1989,  (P.L.  101- 
189)  defines  low-rate  initial  production  of  new  weapons  systems  as  the 
minimum  quantity  necessary  to  (1)  provide  production-configured  or 
representative  articles  for  operational  tests,  (2)  establish  an  initial  pro¬ 
duction  base  for  the  system,  and  (3)  permit  an  orderly  increase  in  the 
production  rate  sufficient  to  lead  to  full  rate  production  upon  successful 
completion  of  operational  testing.  Navy  policy10  uses  a  similar  definition. 
Consistent  with  these  policies  and  definitions,  the  Navy  initially  planned 
to  (1)  approve  what  it  called  limited  rate  production  in  April  1991  for 
four  avionics  systems  (i.e.,  exercise  the  first  contract  option),  (2)  com¬ 
plete  full  operational  testing  in  February  1992,  and  (3)  approve  full  rate 
production  in  April  1992  to  buy  as  many  as  24  more  systems  (i.e.,  exer¬ 
cise  second  and  third  contract  options). 


The  Navy  revised  this  initial  plan  in  response  to  contractor  delays  in 
developing  the  system  and  government  delays  in  providing  certain 
required  government-furnished  equipment.11  Under  the  revised  plan,  the 
Navy  has  delayed  full  operational  testing  by  more  than  2  years  until  the 


9Chief  of  Naval  Operations  Instruction  3960. 10C,  Test  and  Evaluation,  September  1987. 

10Chief  of  Naval  Operations  Instruction  6000.42C,  Research,  Development  and  Acquisition  Proce¬ 
dures,  May  1986. 

1 1  An  enhanced  acoustic  processor  is  being  separately  developed  by  the  Navy  and  will  be  provided  to 
the  Update  IV  contractor  for  integration  with  other  avionics  system  components. 
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first  quarter  of  1994,  but  still  plans  to  buy  the  initial  four  systems  in 
September  1990  (now  calling  this  pilot  production  instead  of  limited  rate 
production)  and  the  additional  24  between  May  1992  and  April  1993 
(now  calling  this  limited  rate  instead  of  full  rate  production).  This 
means  that  the  Navy  plans  to  buy  as  many  as  28  systems  before  it 
knows  whether  the  system  can  satisfy  the  mission  requirement.  When 
questioned,  Navy  program  and  oversight  officials  stated  that  their  pri¬ 
mary  reason  for  increasing  their  initial  rate  quantities  from  four  to  28 
was  the  need  to  exercise  favorable,  fixed-price  contract  options  on  time. 
These  officials  further  stated  that  under  different  contractual  circum¬ 
stances,  they  would  not  buy  so  many  systems  before  completing  full 
operational  testing.  In  our  opinion,  the  Navy’s  decision  to  expand  initial 
rate  production  to  28  systems  is  inconsistent  with  Navy  policy  to  hold 
limited  rate  production  at  minimum  levels  before  completing  full  opera¬ 
tional  testing,  and  consequently  creates  an  unacceptable  risk  of  buying  a 
large  number  of  systems  that  may  not  be  mission  effective. 

We  do  not  support  the  Navy’s  revised  plan,  and  are  concerned  that 
without  the  results  of  full  operational  testing,  the  Navy  will  not  have 
the  information  it  needs  to  make  a  prudent  management  decision  on 
whether  to  buy  more  than  the  initial  four  systems.  Moreover,  given  that 
the  revised  plan  calls  for  delivery  of  twelve  avionics  systems  to  an  oper¬ 
ational  squadron  at  the  same  time  operational  testing  is  occurring,  the 
Navy  risks  sending  systems  to  the  field  that  do  not  work. 

Navy  officials  also  contend  that  they  need  to  exercise  production 
options  on  schedule  to  maintain  price  guarantees,  adding  that  costs 
could  increase  significantly  if  the  contract  is  renegotiated.  However, 
their  estimates  of  cost  increases  are  not  supported  by  detailed  analysis. 
Although  the  Navy  began  an  analysis  to  better  estimate  the  effect  of 
contract  renegotiation  on  costs,  this  analysis  was  not  completed  because 
of  difficulties  in  obtaining  requisite  data  from  the  contractor.  Addition¬ 
ally,  the  Navy’s  estimates  fail  to  recognize  the  substantial  time  and 
expense  that  could  be  incurred  if  an  avionics  system  is  delivered  to  the 
fleet  with  extensive  hardware  and  software  problems.  As  we  previously 
reported,12  it  can  be  six  to  ten  times  more  costly  to  correct  a  software 
discrepancy  after  a  system  is  placed  in  operation  than  it  is  during  early 
system  development.  Also,  even  though  the  Navy  program  manager 
stated  that  it  would  be  difficult  to  renegotiate  the  contract  to  allow  the 
Navy  to  exercise  production  options  after  full  operational  testing,  the 


12Embedded  Computers:  Navy  Not  Ready  to  Buy  Avionics  Computers  for  Its  LAMPS  Mk  I  Helicopters 
(GA0/1MTEC-90-54,  May  31, 1990). 
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Navy  has  recently  negotiated  extending  both  the  third  and  fourth  pro¬ 
duction  options  in  the  Update  IV  contract  by  6  months  at  no  additional 
cost  to  the  government. 

Last,  program  officials  stated  that  their  current  plan  allows  for  ade¬ 
quate  testing  before  making  production  decisions,  noting  that  they  have 
extended  the  initial  phases  of  planned  operational  testing  by  several 
months.  However,  this  extended  testing  will  still  not  support  a  full  rate 
production  decision  because  it  will  only  assess  “potential”  system  effec¬ 
tiveness.  According  to  Navy  policy,  such  testing  only  provides  an  ade¬ 
quate  basis  for  producing  limited  rate  production  quantities.  It  does  not 
constitute  full  operational  testing,  which  is  necessary  to  demonstrate 
system  mission  effectiveness  and  is  required  to  approve  full  rate  pro¬ 
duction.  Moreover,  this  extension  of  initial  operational  testing  will  use  a 
model  of  one  of  the  avionics  system  processors  that  is  not  an  accurate 
representation  of  the  version  of  this  processor  to  be  produced  (see  next 
section). 


Processor  Model  Not 
Representative 


An  enhanced  acoustic  processor13  is  being  separately  developed  and  will 
be  given  to  the  Update  IV  contractor  for  integration  with  the  other  avi¬ 
onics  system  components.  Originally,  the  Navy  was  to  provide  the 
processor  in  April  1990.  However,  hardware  and  software  problems 
have  delayed  the  processor’s  development,  causing  its  delivery  to  the 
Update  IV  contractor  to  slip  to  November  1990.  According  to  program 
officials,  acoustic  processing  is  the  most  critical  function  that  the 
Update  IV  Program  will  perform. 

Because  of  delays  in  developing  this  processor,  initial  operational  testing 
to  determine  whether  to  exercise  the  second  contract  option  will  use  a 
Navy-provided  model  of  the  processor.  The  validity  of  the  initial  opera¬ 
tional  testing  then  will  depend  on  how  accurately  the  model  simulates 
the  acoustic  processor.  However,  the  model  the  Navy  has  provided  the 
contractor  has  less  functionality  and  performance  capability  than  the 
final  version  of  the  processor,  and  thus  is  not  representative.  For 
example,  this  model  can  accept  only  about  one  third  of  the  data  inputs 
to  the  processor.  According  to  the  Navy’s  1986  risk  assessment  for  the 
Update  IV  Program,  use  of  this  model  is  a  high  risk. 


13This  processor,  designated  the  Enhanced  Modular  Signal  Processor,  analyzes  data  from  acoustic 
sensors  to  determine  the  location  and  identity  of  enemy  submarines.  It  offers  improved  acoustic 
processing,  including  more  input/output  channels,  processing  power,  and  operator  aids. 
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Program  Oversight 
Has  Occurred 


Conclusions 


Within  the  Department  of  Defense,  the  management  level  responsible 
for  overseeing  a  system  acquisition  is  generally  determined  by  how 
much  the  system  costs.  For  the  Update  IV  Program,  the  Office  of  the 
Secretary  of  Defense  has  delegated  oversight  to  the  Office  of  the  Secre¬ 
tary  of  the  Navy.  According  to  an  Office  of  the  Secretary  of  Defense 
official,  the  Update  IV  Program  was  delegated  because  the  number  of 
programs  the  Office  of  the  Secretary  of  Defense  oversees  had  to  be 
reduced  to  a  manageable  level. 

In  light  of  the  importance  of  computer  technology  to  the  Update  IV  Pro¬ 
gram,  Navy  oversight  authorities  appear  to  have  focused  on  this  tech¬ 
nology  in  their  program  reviews.  We  found  briefing  documents  involving 
these  oversight  authorities  and  the  program  office  which  show  that 
such  issues  as  completion  of  adequate  contractor  laboratory  test  facili¬ 
ties,  completion  of  the  enhanced  acoustic  processor,  and  status  of 
software  development  have  received  attention.  Additionally,  the  pro¬ 
gram  office’s  current  approach  to  developing  the  Update  IV  Program 
has  been  reviewed  and  approved  by  the  Navy’s  Program  Executive 
Officer  (Acquisitions).  Office  of  the  Secretary  of  the  Navy  review  of  this 
plan  was  scheduled  for  late  August  1990,  but  has  been  delayed  for  sev¬ 
eral  months.  Should  this  final  approval  be  granted,  however,  Navy  over¬ 
sight  authorities  will  be  allowing  the  continued  development  of  software 
without  approved  specifications  and  the  planned  purchase  of  28  sys¬ 
tems  before  full  operational  testing  is  completed.  Thus,  these  authorities 
will  not  be  acting  to  ensure  adherence  to  Navy  software  development 
policies  as  discussed  earlier.  According  to  an  official  in  the  Office  of  the 
Assistant  Chief  of  Naval  Operations  (Air  Warfare),  this  software  devel¬ 
opment  and  testing  approach  has  thus  far  been  approved  because  they 
believe  that  program  costs  will  significantly  increase  and  jeopardize  the 
program  if  contract  production  options  are  not  exercised  on  time. 


The  Navy  is  faced  with  a  difficult  decision  in  acquiring  a  new,  techni¬ 
cally  challenging  avionics  computer  system  for  its  patrol  aircraft.  If  the 
Navy  proceeds  according  to  its  current  acquisition  plan,  it  will  be  unable 
to  perform  thorough  operational  testing,  and  could  therefore  buy  and 
deploy  an  expensive  system  that  does  not  meet  mission  requirements. 

On  the  other  hand,  if  the  Navy  renegotiates  the  contract  to  delay  pro¬ 
duction  decisions  until  it  has  tested  the  system  and  assures  that  it  meets 
operational  requirements,  acquisition  costs  might  increase.  The  Navy 
has  done  no  detailed  analysis,  however,  to  assess  the  extent  of  these 
potential  cost  increases. 
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The  Navy  plans  to  (1)  continue  developing  software  (i.e.,  coding)  before 
approving  detailed  requirements  for  the  system,  (2)  exercise  contract 
options  on  time  and  assume  an  unknown  level  of  risk  that  the  system  it 
buys  may  not  work,  and  (3)  use  a  model  of  an  enhanced  acoustic 
processor  during  testing  that  is  not  representative  of  the  final  version. 

In  our  opinion,  the  Navy’s  current  acquisition  approach  is  unacceptable, 
causing  the  Navy  to  spend  almost  $500  million  on  a  system  that  may  not 
meet  mission  requirements. 


Recommendations 


We  recommend  that  the  Secretary  of  the  Navy  direct  the  Commander, 
Naval  Air  Systems  Command  to  (1)  halt  further  software  development 
(i.e.,  coding)  until  system  specifications  are  approved,  (2)  thoroughly 
justify  the  need  for  initial  rate  production  to  exceed  the  four  systems 
originally  planned,  (3)  conduct  initial  operational  testing  using  actual 
system  components  or  accurate  simulations  of  them,  and  (4)  conduct  full 
operational  testing  before  making  a  full  rate  production  decision.  In  light 
of  the  possibility  that  this  may  preclude  the  Navy  from  exercising 
existing,  fixed-price  contract  options,  we  also  recommend  that  the  Secre¬ 
tary  direct  the  Commander  to  thoroughly  analyze  the  cost  impact  of 
contract  renegotiation,  and  based  on  this  analysis  decide  whether  the 
entire  Update  IV  Program  is  financially  viable. 


As  requested  by  your  offices,  we  did  not  obtain  official  agency  com¬ 
ments  on  a  draft  of  this  report.  However,  we  discussed  its  contents  with 
Navy  and  Office  of  the  Secretary  of  Defense  officials,  and  have  incorpo¬ 
rated  their  comments  where  appropriate.  Our  work  was  performed 
between  December  1989  and  July  1990,  in  accordance  with  generally 
accepted  government  auditing  standards. 

As  arranged  with  your  offices,  unless  you  publicly  announce  the 
report’s  contents  earlier,  we  plan  no  further  distribution  until  30  days 
from  the  date  of  this  letter.  At  that  time,  we  will  send  copies  to  the 
Chairman,  Senate  and  House  Appropriations  Committees;  the  Secre¬ 
taries  of  Defense  and  Navy;  and  to  other  interested  parties.  We  will  also 
make  copies  available  to  others  upon  request.  This  report  was  prepared 
under  the  direction  of  Samuel  W.  Bowlin,  Director,  Defense  and  Security 
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Information  Svstems,  who  can  be  reached  at  (202)  275-4649.  Other 
major  contributors  are  listed  in  appendix  II. 


Ralph  V.  Carlone 
Assistant  Comptroller  General 
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Objectives,  Scope,  and  Methodology 


In  October  1989,  the  Subcommittee  on  Legislation  and  National  Security, 
House  Committee  on  Government  Operations,  expressed  interest  in  the 
Navy’s  plans  to  acquire  embedded  computer  systems  for  selected  anti¬ 
submarine  warfare  systems,  and  asked  that  we  determine  whether  (1) 
the  Navy’s  Update  IV  Program  calls  for  adequate  testing  of  a  new  avi¬ 
onics  computer  system  for  the  P-3  and  P-7  aircraft  before  they  are 
bought,  and  (2)  Navy  management  is  overseeing  the  development  of 
these  embedded  systems.  This  request  relates  to  an  overall  request  from 
the  Chairman  and  the  Subcommittee’s  Ranking  Minority  Member  to 
review  computer  systems  that  are  embedded  in  Defense  weapon 
systems. 

To  accomplish  our  objectives,  we  reviewed  Defense  and  Navy  instruc¬ 
tions  and  standards  governing  the  development,  testing,  and  manage¬ 
ment  oversight  of  embedded  computer  systems.  We  also  reviewed 
Update  IV  documentation  (e.g.,  acquisition  plan,  test  plans,  schedules, 
and  funding  requirements)  as  well  as  the  development/production  con¬ 
tract  and  related  documents  for  the  avionics  program.  Additionally,  we 
interviewed  both  program  officials  responsible  for  managing  software 
development  and  laboratory  officials  responsible  for  monitoring  con¬ 
tractor  performance.  We  also  interviewed  officials  at  the  Navy  test 
activities  participating  in  development  and  operational  testing,  and 
toured  contractor  software  development  facilities. 

Further,  we  interviewed  officials  in  the  Office  of  the  Secretary  of 
Defense  and  the  Chief  of  Naval  Operations  responsible  for  program 
oversight,  and  reviewed  documentation  associated  with  the  discharge  of 
this  oversight  responsibility.  This  work  focused  on  whether  and  to  what 
extent  this  oversight  specifically  addressed  the  embedded  avionics  com¬ 
puter  system. 

We  performed  our  work  between  December  1989  and  July  1990,  prima¬ 
rily  at  the  Update  IV  program  office  within  the  Naval  Air  Systems  Com¬ 
mand,  Arlington,  Virginia,  and  the  Naval  Air  Development  Center, 
Warminster,  Pennsylvania.  We  also  visited  the  Naval  Air  Test  Center, 
Patuxent  River,  Maryland;  the  Navy’s  Operational  Test  and  Evaluation 
Force,  Norfolk,  Virginia;  and  the  contractor’s  system  development  facili¬ 
ties  in  Seattle,  Washington. 

As  requested  by  the  Chairman’s  office,  we  did  not  obtain  official  agency 
comments  on  a  draft  of  the  report.  However,  we  discussed  its  contents 
with  Navy  and  Office  of  the  Secretary  of  Defense  officials,  and  have 
incorporated  their  comments  where  appropriate.  We  conducted  our 
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review  in  accordance  with  generally  accepted  government  auditing 
standards. 
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Management  and 
Technology  Division, 
Washington,  D.C. 


John  B.  Stephenson,  Assistant  Director 
Randolph  C.  Hite,  Assignment  Manager 


Philadelphia  Regional 
Office 


Norman  C.  Berman,  Evaluator-in-Charge 
Amy  Ganulin,  Evaluator 
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